Multiple Comparisons: Bonferroni Corrections and False Discovery Rates

Statistical analysis of a data set typically involves testing not just a single hypothesis, but rather many (often very many!). For any particular test, we may assign a pre-set probability α of a type-1 error (i.e., a false positive, rejecting the null hypothesis when in fact it is true). The problem is that using a (say) value of α = 0.05 means that roughly one out of every twenty such tests will show a false positive (rejecting the null hypothesis when in fact it is true). Thus, if our experiment involves performing 100 tests, we expect 5 to be declared as significant if we use a value ofα = 0.05 for each. This is the problem of multiple comparisons, in that we would like to control the false positive rate not just for any single test but also for the entire collection (or family) of tests that makes up our experiment.